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THE NEHARI MANIFOLD FOR FRACTIONAL SYSTEMS 
INVOLVING CRITICAL NONLINEARITIES 

XIAOMING HE, MARCO SQUASSINA, AND WENMING ZOU 


Abstract. We study the combined effect of concave and convex nonlinearities on the number of positive 
solutions for a fractional system involving critical Sobolev exponents. With the help of the Nehari 
manifold, we prove that the system admits at least two positive solutions when the pair of parameters 
(A, /r) belongs to a suitable subset of R^. 
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This paper is concerned with the multiplicity of positive solutions for the following elliptic system 
involving the fractional Laplacian 

{ (-A)*m = \\u\^-‘^u + -^\u\°^-'^u\vf in fl, 

(—A)®t; = -\- -^^\u\°‘\v\^~'^v in fl, 

u = V = Q on 


where C is a smooth bounded domain, A, /t>0, 1 < q <2 and a > l,/3 > 1 satisfy a + {3 = 2* = 
2N/{N — 2s), s € (0,1) and N > 2s. When Qf = /3,a-|-/3=_p<2*,A = /t and u = v, problem (1.1) 
reduces to the semilinear scalar fractional elliptic equation 


( 1 . 2 ) 


(—A)®u = Alul"? in fl, 

M = 0 on do,. 


Recently, a great attention has been focused on the study of nonlinear problems like (1.2) which involve 
the fractional Laplacian. This type of operators naturally arises in physical situations such as thin obstacle 
problems, optimization, population dynamics, geophysical fluid dynamics, mathematical finance, phases 
transitions, straitified materials, anomalous diffusion, crystal dislocation, soft thin films, semipermeable 
membranes, flames propagation, conservation laws, ultra-relativistic limits of quantum mechanics, quasi- 
geostrophic flows, multiple scattering, minimal surfaces, materials science and water waves, see [21]. We 
refer to [12,23,26,27,29,32] for the subcritical case and to [3,4,11,14,15,22,24,28] for the critical case. 
In the remarkable paper [9], Caffarelli and Silvestre gave a new formulation of the fractional Laplacian 
through Dirichlet-Neumann maps. This is extensively used in the recent literature since it allows to 
transform nonlocal problems to local ones, which permits to use variational methods. For example, 
Barrios, Colorado, de Pablo and Sanchez [3] used the idea of the s-harmonic extension and studied the 
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effect of lower order perturbations in the existence of positive solutions of (1.2). Brandle, Colorado and 
de Pablo [5] investigated the fractional elliptic equation (1.2) involving concave-convex nonlinearity, and 
obtained an analogue multiplicity result to the problem considered by Ambrosetti, Brezis and Cerami 
in [2]. In the case q = 2 and p = 2*, Servadei and Valdinoci [24] studied (1.2) and extended the classical 
Brezis-Nirenberg result [ 6 ] to the nonlocal case. In [ 8 ], Cabre and Tan defined (—A)^/^ through the 
spectral decomposition of the Laplacian operator on fl with zero Dirichlet boundary conditions. With 
classical local techniques, they established existence of positive solutions for problems with subcritical 
nonlinearities, regularity and T°°-estimates for weak solutions. In particular. Tan [28] considered 


(1.3) 


(-A)i/2u= Au-f 
u = 0 


in fl, 
on dil, 


investigating the solvability (see also [32] for a subcritical situation). Very recently, Colorado, de Pablo, 
and Sanchez [14] studied the following nonhomogeneous fractional equation involving critical Sobolev 
exponent 

j{—Ayu=\u\'^‘~'^u + f{x) in fl, 

= 0 on dfl. 

and proved existence and multiplicity of solutions under appropriate conditions on the size of /. For the 
same problems, Shang, Zhang and Yang [22] obtained similar results. 

The analogue problems to (1.1) for the Laplacian operator have been studied extensively in recent years, 
see [1,13,19,20,30,31] and the references therein. In particular, Hu and Lin [20] studied the Laplacian 
system with critical growth and obtained the existence and multiplicity results of positive solutions by 
variational methods. 


The purpose of this paper is to study system (1.1) in the critical case a + (3 = 2*. Using variational 
methods and a Nehari manifold decomposition, we prove that system ( 1 . 1 ) admits at least two positive 
solutions when the pair of parameters (A,/r) belongs to a certain subset of To our best knowledge, 
there are just a few results in the literature on the fractional system ( 1 . 1 ) with both concave-convex 
nonlinearities and critical growth terms. We point out that we adopt in the paper the spectral (or 
regional) definition of the fractional laplacian in a bounded domain based upon a Caffarelli-Silvestre type 
extension (see [14]), and not the integral definition. We shall refer to [25] for a nice comparison between 
these two different notions. In [18], a problem like (1.1) with g = 2 is investigated, using the integral 
notion, from the point of view of existence, nonexistence and regularity. 


To formulate the main result, we introduce 


(1.4) 


Ai := 



{ksS{s,N))-i\n\ 



2 

2-q 


2-g 

.2(2J-g) 


2 * 


2 


2 _ 

^ -2 


where jUj is the Lebesgue measure of fl, fcg is a normalization constant and S{s, N),Ss^a,i 3 are best 
Sobolev constants that will be introduced later. For 7 > 0, we also consider 


:= {(A,/r) e M+ : 0 < A^-s -h < 7 }. 


Then we have the following 


Theorem 1.1. The following facts holds 

{i) system (1.1) has at least one positive solution for all {X, p) € %_!. 

(ii) there is A 2 < Ai such that (1.1) has at least two positive solutions for (A,/r) € ‘^A 2 - 


Concerning regularity, one can get a priori estimates for the solutions to (1.1) and hence obtain, as 
in [3, Proposition 5.2], that u,v £ for s = 1/2, u,v £ C°’^®(H) if 0 < s < 1/2 and u,v £ 

if 1/2 < s < 1 . 

The paper is organized as follows. In Section 2 we introduce the variational setting of the problem and 
present some preliminary results. In Section 3 we show that the Palais-Smale condition holds for the 
energy functional associated with (1.1) at energy levels in a suitable range related to the best Sobolev 
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constants. In Section 4 we give some properties about the Nehari manifold and fibering maps. In Section 
5 we investigate the existence of Palais-Smale sequences. In Section 6 we obtain solutions to some related 
local minimization problems. Finally, the proof of Theorem 1.1 is given in Section 7. 

2. Some preliminary facts 

In this section, we collect some preliminary facts in order to establish the functional setting. First of 
all, let us introduce the standard notations for future use in this paper. We denote the upper half-space 
in by 

:= {z = (x,y) = (xi,--- ,XN,y) € : y > 0}. 

Let il C be a smooth bounded domain. Denote by 

Cn := D X (0,oo) C 

the cylinder with base D and its lateral boundary by OlCq := dil x (0, oo). The powers (—A)'* of the 
positive Laplace operator —A, in D, with zero Dirichlet boundary conditions are dehned via its spectral 
decomposition, namely 

CO 

{-Ayu{x) :='^ajPjipj{x), 

where is the sequence of eigenvalues and eigenfunctions of the operator — A in D under zero 

Dirichlet boundary data and aj are the coefficients of u for the base in In fact, the 

fractional Laplacian (—A)® is well defined in the space of functions 

CO OO ^^2 

iL(5(D) := (u = G : Mh^ = < ooj, 


j=i 


i=i 


and ||rt||iL« = ||(—The dual space H ®(D) is defined in the standard way, as well as the 
inverse operator (—A)“'*. 

Definition 2.1. IFe say that {u,v) G 77o(D) x Hq{^) is a solution of (1.1) if the identity 

f + {-A)^v{-A)i(p 2 ) dx - f + p\v\‘^~‘^vip 2 )dx 

Jn Jn 

-^ [ \uy~'^u\vfipidx -^ f \uy\vf~'^v(p 2 dx = 0 , 

a + p Jq a + P Jq 

holds for all (</?!, (^ 2 ) G Hoi^) ^ 

Associated with problem (1.1), we consider the energy functional 

^ / (l(-A)^up-f |(-A)4up) da; - - [ {X\u\‘> + p,\v\‘^)dx 
^ an dan 


a + P 


I 

an 


;|^da:. 


The functional is well defined in iLo(D) x i7g(D), and moreover, the critical points of the functional 
correspond to solutions of (1.1). We now conclude the main ingredients of a recently developed technique 
used in order to deal with fractional powers of the Laplacian. To treat the nonlocal problem (1.1), we 
shall study a corresponding extension problem, which allows us to investigate problem (1.1) by studying 
a local problem via classical variational methods. We first define the extension operator and fractional 
Laplacian for functions in i7o(^) ^ ^o(^)- We refer the reader to [3-5,10] and to the references therein. 

Definition 2.2. For a function u G we denote its s-harmonic extension w = Es{u) to the cylinder 

Cq as the solution of the problem 

div(y^“^®Vu>) =0 in Cq, 
w = 0 on OlCq 

= u on Ct X {0}, 
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and 

i-Ayu{x) = -ks lim 

y^ 0 + ay 

where kg = — s)/r(s) is a normalization constant. 

The extension function w{x, y) belongs to the space 

X^yCn) :=Co-(f)x [ 0 ,oo))"'"^«‘"^’ 

endowed with the norm 




1/2 


The extension operator is an isometry between i7g(f2) and XQ(Cn), namely 

l|M||ff 5 (n) = ll-E^s(M)||xg(Cs,), for all u € 

With this extension we can reformulate (1.1) as the following local problem 
—div( 2 /^“^®Vwi) = 0, —div( 2 /^“^®Vw 2 ) = 0 in Cq 


( 2 . 1 ) 


where 


Wi = W2 = 0 
Wi = U, W2 = V 

^ + ^^\w,y-^w,\w2\y 

dw2 _ 


= /i|w 2 |'^ + J^|wi|“|w 2 |^ ‘^W2 


on OlCq 
on fl X {0} 
on n X {0} 
on n X {0}, 


§11 to to 1 . 2 , 

y^o+ oy 

and wi,W 2 G Xq{Cq) are the s-harmonic extension of u,z; G respectively. Let 

E^iCn) := X^iCn) x XJ(Cn). 

An energy solution to this problem is a function (u;i,ty 2 ) G Eq{Cq) satisfying 


h'- 

JCn 


®Vu;i • X(pidxdy + kg 


= X 


[ kil 

in 

E j 

in 


‘^wi(pidx + 
'^~^W2^2dx 


/ y^ ^^Vw2 ■ Xip2dxdy 
JCn 

[ \wi\°‘~‘^wi\w2fyJidx 
in 

[ \wiy\w2f~‘^W2(p2dx, 
Ja 


’ Cn 
2a 

a + fi 

a + P 


for all {(pi,Lp 2 ) G AQ(Cn). If {wi,W 2 ) G A'g(Cn) satisfies (2.1), then {u,v) = (wi(-,0), W 2 (-,0)), defined in 
the sense of traces, belongs to the space LIo(^) ^ ^o(^) ^ solution of the original problem (1.1). 

The associated energy functional to the problem (2.1) is denoted by 

■=I\,^.iiwi,W2) = ^ [ 2/^~^*(|Vwi|^ + \Xw2\'^)dxdy 
^ JCn 


-f 

q in 


(A|wi|'^ + y\w2\'^)dx — 


a + fd 


I 

in 


\w 1 y\w 2 fdx. 


Critical points of Ia.^i in Eq{Cq) correspond to critical points of Jx^y, : Hq{V,) x iLo(^) 
following lemma we list some relevant inequalities from [5]. 


In the 


Lemma 2.3. For any 1 < r < 2* and any z G Xq{Cq), it holds 

2 

(2.2) \u{x)\^dx^ < Cy y^~'^‘‘\Xz{x,y)\'^dxdy, u:=Tr{z), 

for some positive constant C = C{r,s,N,i}). Furthermore, the space Xq{Cq) is compactly embedded into 
L’’(0), for every r < 2*. 
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Remark 2.4. When r = 2*, the best constant in (2.2) is denoted by S{s, N), that is 

JCn 


(2.3) 


Sis, N) •= inf 

^eA 5 (Cs,)\{o} 


(f |z(a;, 0 )p“dx) 

J Q 


It is not achieved in any bounded domain and, for all z £ X®(]R(^“''^), 
(2.4) - - / /■ 


N-2s 

Iv 


f z{x,y)'\^dxdy > S{s,N) ( f |z(a;, 

Js.'l+^ \Jk^ 

S{s, N) is achieved for 17 = R'^ by functions Wg which are the s-harmonic extensions of 

^{N-2s)/2 


(2.5) 


.{x) := 


(£2 + | 2 .| 2 )(A- 2 .)/ 2 ^ 


£ > 0 , X G 


Let U{x) = (1 + |xp) and let W be the extension of U (cf. [3,5]). Then 


yV{x,y) = Es{U) = CN,sy^‘' [ 

Jr 


U{z)dz 


,, n n^K±I£ ’ 

R'V (|x — zp + y‘‘) 2 

is the extreme function for the fractional Sobolev inequality (2.4). The constant S{s,N) given in (2.3) 
takes the exact value 


S{s,N) = 


2^sr(^^)r(i-s)(r(f))^ 

r{s)T{^){r{N)y 


and it is achieved for 17 = R'^ by the functions w,.. 

Now, we consider the following minimization problem 

[ y^-^^{\Vwi\‘^ + \Vw2\‘^)dxdy 
JCn 


( 2 . 6 ) 


^s,oi,p •— 

(ttJi ,W2)^E^{Cn)\{0} 


(/ |u;ir|u; 2 |''dx)' 

J r 2 


Using ideas from [1], we establish a relationship between S{s,N) and 5s,a ./3 (see also [18]). 
Lemma 2.5. For the constants S{s,N) and Ss^a,i 3 introduced in (2.3) and (2.6), it holds 


(2.7) 


5s 


a \ 


S{s,N). 


In particular, the constant Ss,a,p is achieved for O = . 


Proof. Let {z„} C Xq(Cq) be a minimization sequence for S{s,N). Let cr,i > 0 to be chosen later and 
consider the sequences wi^n ■= crzn and W 2 ,n ■= tzn in Xq(Cq). By means of (2.6), we have 


Jcn 


[ ^lVz„(x,?/)l^dxdy 
JCn 


(a°^t^) ■ 


\zr,\ -dx 


— Ss^a,t 


2/3 


Defining g : R+ —?> R+ by setting g{x) := x^ + we have 

a^+e 


{a°‘tP) '■ 


^(t)’ = ci(xo) =g(\/c^) = + 
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Choosing cr, t in the previous inequality such that a jt = \fajp and letting n —>■ oo yields 


a \ ^ 


S{s, N) > Ss^a,P- 

On the other hand, let {(wi,n, W2,n)} C £'q(Cq)\{ 0} be a minimizing sequence for Ss,a,i3- Set hn ■= <JnW2,r 
for (T„ > 0 with \wi^n\'^‘^dx = \hn\'^‘'dx. Then Young’s inequality yields 

[ Iwi^nl^^lhnfdx < -^ [ \wi^n\'^‘’dx + ^ f \hn\'^‘‘dx = f \hn\'^‘‘dx= [ \wi^n\'^‘’dx. 

JJVL JO. J Q J Q 


In turn, we can estimate 


' Cn 


y)p + \Ww2,n{x,y)\'^)dxdy 


( j \u)l,n\°‘\w2,nfdx'^^ 

[ Vwi,„(x, 2 /)p + |Vw 2 ,n(a;, y)\'^)dxdy 

JCn 


20 


> 


( J \wi^n\°'\hnfdx'j ' 

^ f 
JCn 


2/2* 


20 

W —2 


\wi^n\'^^dx 

r 2 z 

[ y^~'^'‘\^hn{x,y)\'^dxdy 

JCn 


\hn\'^-dx) 

J 


2/2* 


> S{s,N)g{an) > S{s, N)g{y/a/P). 
Passing to the limit as n —^ oo in the last inequality we obtain 

'P 


a \ ^ 


S(^S,N) < Ss,a,fi- 

Whence, the conclusion follows by combining the previous inequalities. 

In the end of this section, we fix some notations that will be used in the sequel. 


□ 


Notations. In this paper we use the following notations: 

• l<p<oo denote Lebesgue spaces, with norm || • ||p. E = Xq{Cq) x Xg(Cn) is equipped 

with the norm = ||(wi,u; 2 )|p = + Ik 2 |lx-(c^)- 

• The dual space of a Banach space E will be denoted by E~^. We set tz = t{wi,W 2 ) = {twi,tw 2 ) 
for all z & E and t G K. z = {wi,W 2 ) is said to be non-negative in Cn if tci(x, y) > 0, W 2 [x, y) > 0 
in Co and to be positive if wi(x, y) > 0, W 2 {x, y) > 0 in Cq. 

• |fl| is the Lebesgue measure of fl. B{0; r) is the ball at the origin with radius r. 

• 0(e*) denotes |(!I(e*)|/e* < C as £ —)► 0 for f > 0. o„(l) denotes o„(l) —>■ 0 as n —oo. 

• C,Ci,c will denote various positive constants which may vary from line to line. 


3. The Palais-Smale condition 

In this section we shall detect the range of values c for which the (PS')c-condition holds for the 
functional X\^^. Let c G K and set, for simplicity, E := EQ{Cn). We say {zn} C i? is a (P«S')c-sequence in 
E for lA,/i if X\^^{zn) = c-|-o„(l) and ^(-Zn) = On(l) strongly in E~^, as n —>■ oo. If any (PS')c-sequence 
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{zn} in E for admits a convergent subsequence, we say that satisfies the (P5')c-condition. We 
shall need the following preliminary result. 


Lemma 3.1. Let {z„} (Z E he a {PS)c-sequence for for some c € K. with Zn ^ z in E. Then 
^{z) = 0 and there exists a positive constant Kq, depending only on q,N,s and |n|, such that 


Proof. Consider Zn = {wi^n,W 2 ,n) C E and z = {wi,W 2 ) € E. If {Zn} is a (PS')c-sequence for 
with Zn ^ z in E, then wi^n wi and W 2 ,n W 2 in Xq{Cq), as n —>■ oo. Then, by virtue of Sobolev 
embedding theorem (Lemma 2.3), we also have Wi^n(z 0) Wi(-, 0) and W 2 ^n(z 0) W 2 (-, 0) strongly in 

as n —?► oo. Of course, up to a further subsequence, r(;i_„(-,0) —?► 'u;i(-,0) and 7C2,ra(‘,0) —>■ W2(-,0) 
a.e. in fl. It is standard to check that T'^ ^{z) = 0. This implies that {I'^^^{z),z) = 0, namely 

ks [ {\Vwi\'^ + \Vw 2 \^) dxdy = f (A|wi|'^ ++ 2 f |r(;i|“|r(; 2 |^dir- 

J Cci J Vi J Cl 

Consequently, we get 

(3.1) Ix.f,{z) = (^^- y^"^®(|Vw;iP + |Vw2ndxdy 

-(-“ 4 ) [ + ii\w2\‘‘)dx. 

By (3.1), Holder and Young inequalities and the Sobolev embedding theorem, we obtain 


kxA^)= + T\w2ndx 




q2 


^|H|K-9)/2:(^^5(,^jV))-5(A||n;ir + /r||R;2r) 




2-q 


A2-5 -|_ ^2- 


|e [Ikif 


W2 


= -Kq +/i^) , 

which yields the assertion, where we have put 

C := ?i^|H|(2:-9)/2:(fc,5(s,iV))-f, e := ( 


q^t 


K 2s J 


Ko := 


the positive constants involving only q, |H|, s and N. 


□ 


Lemma 3.2. If {zn} G E is a {PS)c-sequence for X\^^, then {Zn} is bounded in E. 


Proof. Let = (wi_„,ty 2 ,n) C if be a (PiS')c-sequence for Xx ^j^ and suppose, by contradiction, that 
||z„|| —?► oo, as n —?► oo. Put 


Zn = (Wl.n, W2.n) 


Zn _ /Wl,n W2,n\ 

\\Zn\\~y\\Znr\\ZnV- 


We may assume that Zn ^ z = ( 1 / 11 , 1 / 12 ) in E. This implies that /i’i,n(‘i0) —>■ //;i(-,0) and ///2,n(‘i0) 
u; 2 (-, 0 ) strongly in L’’(H) for all 1 < r < 2* and, thus. 


(A|/Cl,n|'^ + fl\w 2 ,n\'^)dx = [ (A|/(;i|'^ + p.\w 2 \'^)dx + 0„(1). 

2 JQ 
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Since {zn} is a {PS)c sequence for and ||z„|| —>■ oo, we get 


(3.2) 

ks 

y 

JCn 


+ |Vw2,nP 

9 

/ (Akl.nl'^ + Mk2,n 

JQ 

\'^)dx 


- 

2|kn||^^ 

2* 

[ kl.nl 

JQ 

“k2,nl^rf2; 

= 0^(1), 



and 







(3.3) 

ks 

I 

JCn 


+ |Vl(;2,n|^ 

') dxdy - kn 1'^“^ 

/ (Akl.nl'^ + Mk2,n| 
IQ 

'^)dx 


- 


f kl.nl 

JQ 

“k2,n|'^rf2; 

= o„(l). 




Combining (3.2) and (3.3), as n —>■ oo, we obtain 

(3.4) k, [ (|Vzfii,„|2 + \Vw 2 ,n\^) dxdy 

JCn 

= + ^^\w2,n\^)dx + 0 „( 1 ). 

In view of 1 < g < 2 and ||z„|| —>• oo, (3.4) implies that 

ks [ (iVwpnp + |Vw 2 ,„P) dxdy 0, 

JCn 

as n —>■ oo, which contradicts to the fact that \\Zn\\ = 1 for any n > 1. □ 


Lemma 3.3. T\_fi satisfies the {PS)c condition with c satisfying 


—oo < c < Coo := 


2s ^ kgSs^a,fz 

TV 


- Ko 


where Kq is the positive constant introduced in Lemma 3.1 

Proof. Let {zn} C E he a (PS')c-sequence for with c £ (—oo,Coo). Write = {wi^n,W 2 ,n)- By 
Lemma 3.2, we see that {zn} is bounded in E and Zn ^ z = (wi,W 2 ) up to a subsequence and z is a 
critical point of Furthermore, ^ wi and W 2 ,n W 2 weakly in Xq(Cq), wi^ni-,0) —t ti'i(‘, 0) and 
W 2 ,n{-, 0 ) -t W2(-,0) strongly in L’’(fl) for every 1 < r < 2* and wy„(-,0) -)> wi(-, 0), W2,n(-, 0) W2(-,0) 

a.e. in fl, up to a subsequence. Hence, we have 

(3.5) [ {X\wi,n\‘^ + p\w 2 ,n\‘^)dx = f {X\wi\'' + fi\w 2 \‘^)dx + Onil). 

Jq. Jq. 

Let wi,n '■= wi^n — wi, W 2 ,n '■= W 2 ,n — W 2 and Zn '.= {wi,n,W 2 ,n)- Then, we obtain 

||z„f = ||z„f _||^||2 + o„(l). 

In light of [19, Lemma 2.1], we also get 

(3.6) [ |u;i^„|“|w2,n|^fia; = [ \wi^n\°‘\w 2 ,nfdx- [ |wi|“|w2|'^d2; + 0„(1). 

Jq. Jq Jq 

Using I\,fj_{zn) = c + o„(l) and ^(-Zn) = o„(l) and (3.5)-(3.6), we conclude 
(3-7) xll^nll^--;^ / |wi,nr|w2,n|'^rfa; = C-lA,M(^) + 0„(l), 

^ JQ. 

W'ZnW'^ -2 f |wi,„|“|w2,n|'^rfa; = {T'x,^{Zn), Zn) - {P'x,^.{z), z) + 0„(1) = 0„(1). 

JQ 


Hence, we may assume that 
(3.8) 


\wi,n\°'\w2,nf dx -)■ 
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If £ = 0, the proof is complete, li £ > 0 then from (3.8) and the definition of Ss,a,p, we have 

( 7\ ) — ^s^s,a,/3 lim [ f ^ lilTL 

\2J n^c,o\J^ ) n^oo 

which implies that £ > 2(ksSs,aS 1^)^ ■ On the other hand, from Lemma 3.1, (3.7) and (3.8), 


2 2 

which contradicts c < Coo- 




- iLo (a 2 -« , 


□ 


4. The Nehari manifold 


Since the energy functional associated with (2.1) is not bounded on E, it is useful to consider the 
functional on the Nehari manifold 

:={zGi5\{0}: z) = 0}. 

Thus, z = (wi, W 2 ) G A/a,/j if and only if z ^ 0 and 

(4.1) (i;^^(z),z) = ||zf-Q a,m(^)- 2 /" |wi|“|ri; 2 |^da; = 0, 

in 


where 


QxA^) '■= [ (A|wi|'^ + Atk2|'^)dx. 

in 


It is clear that all critical points of must lie on A/a,^ and, as we will see below, local minimizers on 
are actually critical points of Ta./x- We have the following results. 

Lemma 4.1. The energy functional Ix^^ is bounded below and coercive onMx,^- 

Proof. Let z = (wi,W 2 ) G A/a,^. Then by (4.1) and the Holder and Sobolev inequalities 


(4.2) 




9* — 9 9* — n 

° ''z\?-^^QxA^) 


> 


22 * 

2 *- 2 , 
22 ! 


2 - 


<Z 2 ! 

2 ;-g 

g 2 * 


(fc«5(s,iV))-i|H|¥ ( 


A2-<! + /r2-5 


Since 1 < q < 2, the functional lx ,11 is coercive and bounded below on Afx,fj,- 


□ 


The Nehari manifold Afx,n is closely linked to the fibering map $2 : t —>• Ix,tJ.{tz) given by 

H r 9f'^* f 

^^{t) ■.= Xx,^,{tz) = —WzW^ -/ {X\wi\'^ + ^i\w 2 V)dx -— / \wx\‘^\w 2 fdx. 

2 g in a + /3 in 

Such maps were introduced by Drabek and Pohozaev in [16] and later on used by Brown and Zhang [7]. 
Notice that we have 

[ (Alwil"^ + ^|ui2|'’)da; - 2t^'>“^ f |uii|“|ui2|^da;, 

Jq Jq 

®"(^) = ll-^lP - (g - [ (Ajwij'^ + ^|w2|‘^)da; - 2(2* - [ \wi\°'\w2fdx. 

in in 

It is clear that $(,(t) = 0 if and only if tz G Afx,fi- Hence, z G Afx.fj. if and only if $(,(1) = 0. Introduce 
now the functional 


T^x,niz) ■= {^'x,i,{z),z). 
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Then, for every z £ we have 

= 2\\zf-qQx,i,{z)-22l f \wi\°^\w 2 fdx 

JQ. 

(4.3) = (2-g)||zf-2(2*-g) /" \wi\°'\w 2 f dx 

Jn 

= {2:-q)Q,,,{z)-{2t-2)\\zr. 

Following the method used in [30], we split into three parts 

■= ^ > 0}> 

:= {z e ■ (^Am(^)’ = 0}’ 

:= {z e A6v.;. : (7^A,^(z), z) < 0}. 

Then, we have the following lemmas. 

Lemma 4.2. If zq is a local minimizer for on M\^fi and zq ^A/”®^, thenl'^ ^{zq) = 0. 

Proof. Let zq = (u'o.ij ii'0,2) £ be a local minimizer for the functional Ix.^ on ffx,fi- Hence, there 
exists a Lagrange multiplier 7 £ R such that X'^ ^{zo) = iT^'x ^{zq)- Thus, 

i^xAzo), zo) = 7 (^ a ^( 2 o ), zo) = 0 . 

Since zq ^ A/”®^, then {TZ'^ ^(zo),zo) ^ 0, yielding 7 = 0. This completes the proof. □ 


Let Ai be the positive number defined in (1.4). Then we have the following result. 
Lemma 4.3. Assume that (A,/r) £ • Then Mx ^ = 0. 


Proof. Assume by contradiction that there exist A > 0 and /i > 0 with 0 < X^-i + < Ai and such 

that Afx fj, ^ bet z £ A/”^^. Then, by virtue of (4.3), we get 


UlP = 


2(2: - q) 


wi W2rda; 


fdz 


2 — <7 Jn 

By Holder inequality and the Sobolev embedding theorem, we have 

2-q 


- 211 ^ = ^ - IQxAz)- 


2t - 2 


U > 


(2(2 *-g) 

‘21-q 


{kgSg^a,^) 2 


\\z\\<{^^ihSis,N))-^n\^y^^{X^+q^^)K, 


which leads to the inequality 

2 , 2 ^ f2*-q 


A2-, + ^ 2 -^ > _^^ {ksS{s,N)) 

contradicting the assumption. 


2-q 


(2(2 *-g) 


{ksSs^a^js) ^ 


— Ai, 


□ 


From Lemma 4.3, if 0 < A^-® + ^2-, < Ai, we can write Afx.ji = Afxfi '^■^x fi define 
ax,f, ■■= inf Xx,^,{z), a+ := inf Xx,^{z), af := inf Ia.m(^)- 

Moreover, we have the following properties about the Nehari manifold Aa^^. 


Theorem 4.4. The following facts holds 

(i) If (A,/J,) £ '^Ad then we have ax,fi < Oa ^ < 0 ; 

{ii) If (A, 71) £ ‘^(q/2)2/(2-g)Ai; then we have ctf^ > cq for some positive constant cq depending on 
A, 71, N, s and |H|. 
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Proof, (i) Let 2; = {wi,W 2 ) & By formula (4.3), we have 


2-g 

2 ( 2 * - q) 


||z||2> f \w,nw2f 

Jq 


and so, 




zr + 2 -- 


1 1 


q 2* 


s / ^ Q 


< 


1 1 \ 2 -q 


1 1 
2 ~q 
1 1 

2 qj ' \q 2*y 2* - g 


^1 \W 2 \ Ctx 


\^dx 


Ikll^ 


Therefore, by the definition of we can deduce that < 0. 

(ii) Let z £ By equation (4.3), 

2 -q 


-||zf < f \w,nw2fdx. 

in 


2(2*-g)' 

By the Holder inequality and the Sobolev embedding theorem, we have 


Hence, we obtain 


2 > 


/ \wi\°‘\w2fdx<{ksSs,a,l3) 
in 

/ 2 -q 


V2(2*-g) 

From the last inequality we infer that 

2:-2„ 2*-g, 


{ksSs,a, 0 )^^, for all z G Af. 


X,fi' 


A.(^) = ^ii^ir-^QA,,(z) 


> lur 


2: - 2, 


lkf-^-^(fc.5(s,iV))-«|0|¥'(A^+;.^)' 


> 


22 * 
2-g 

mt-q) 

K-q 

92* 


g2 


2 * - 2 , 


{ksSs^a,p)^‘ 00 * iksSs,a,l 3 ) 


( 2 -q)N / 2 — g \ 


L 22 


2 (2 *-g) 


{ksS{s,N))-^n\^ 


Thus, if A^-s +^2-g < (g/2)2-5Ai, then 

d^x,r.{z) > Co, for all z G 
for some positive constant cq = co(A, /r, s, N, |H|). 


□ 


Lemma 4.5. Let (A,/r) G ■ Then, for every z = (wi,W 2 ) G i?\{(0, 0)}, t/iere exist unique numbers 
t~ = t~{z) > 0 and = t'''(z) > 0 such that 


e A/'+^, t-z G 


In particular, we have 


t+ <t<t~ 


t := 


(2* - q)Qx,f,{z) 


^ (2*-2)||z|P J 

as well as 11 -^ Ix,fj,{tz) strictly increasing on [t~^,t~] and 

= min_lA,M(^ 2 )i d:x^f,{t~z) = maxlx,^{tz). 




t>0 
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Proof. Fix 2; = (101,102) £ E, so that Qx^^j.(z) > 0 , and let m : ( 0 ,00) —>• R be defined by 
(4.4) m(t) -t‘^~‘^"Qx,^i(z), for t > 0 . 

Obviously, m(t) —>■ —00 as t —>■ 0 + and m(t) —>■ 0 and m(t) > 0 as t —00. Since 

m'(t) = (2 - ||zf -(q- 

we have m'(t) = 0 at t = t = tmax, Tn'(t) > 0 for t £ (0,tniax) and m'(t) < 0 for t £ (tmax,oo). Hence, 
m achieves its maximum at tmax, is increasing for t £ ( 0 ,tinax) and decreasing for t £ (tmax, 00). By 
(X,fi) £ ‘^Ai, (4.2) and (1.4), we have 


QxAz)<{ksS(s,N))-i\n{-^ ' ||z|| 

< (hS(s,N))-i\^-^ 


(4.5) 


2 ! - 2 , 


2-g 
2(2* - q) 


21-q" 

By (4.5) and a simple calculation we have 

m(tinax) = Qx^^{z) 


{ks^S^OL,^^ 2 


2 T^ 


( 2 * -q)Q\A^) 


L ( 2 J- 2 )||z||^ 


2 - 2 * 

“ 2 ^ 


- 


(2* - q)Qx,^(z) 


L (2j-2)|k|| = 


Qx,ti.(z) 


(4.6) 


0* _ n\ 2-q O _ 2(2t-q) 

* Xz-^\\z\\-^QxA^)^ 


21-2 


2 *-<z' 


> 2\\z\\'^‘’(ksSs,a,p) 

Then, taking into account the definition of Sg^a.p we have 


m( 0 ) = -00 < 0 < 2 / \wi\°'\w 2 fdx < 2(ksSs,a,l3) ^ \\z\\'^‘ < m(tmax). 

Jn 

In turn, there exist unique t+ and t~ such that 0 < f*' < tmax < t“, 

m(f'") = 2 f |?iii|“|w 2 |^da; = m(t“), 

Jq 

and m'(t'*') > 0 > m'(t~). By the equation for ‘I’(,(t) and (4.4) we have 

(4.7) $;(t) = t^:-! [m(t) - 2 [ |n;inn; 2 |^dJ , 

L Jn 

which yields $(,(t^) = 0. By virtue of (4.7), we also have ±$"(t^) > 0. This shows that $2 has 
a local minimum at T*" and local maximum at t~ with t^z £ function t i-A Ix,fi(tz) is 

increasing on [t''",t“] and decreasing over [0,t+] U [t~,oo). Hence, Ix,iJ.it~^z) = mino<t<t-TA,/i(t 2 ) and 
d^x.ii(t~z) = maxt>oTA,/i(tz), concluding the proof. □ 


5. Existence of Palais-Smale sequences 

Lemma 5.1. Let (X,fj,) £ . Then, for any z £ Xfx.p., there exists r > 0 and a differentiable map 

: B(0-, r) C E —>■ R+ such that ^(0) = 1 and £,{h)(z — h) G Xfx,fi for every h £ H(0; r). Let us set 

Ti := 2ks / (Vwi • Vhi + Vw 2 • Vti 2 ) dxdy, 

JCn 

72 := 9 / {X\wi\‘^~'^wihi + p\w 2 \'^~'^W 2 h 2 ) dx, 

Jq. 

Ts := 2 /" (Q;|?iii|““^wihi|w2|^ +/3|wi|“|?i'2|^“^W2/i2) da;, 

Jq 
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for all {hi,h 2 ) € E and {wi,W 2 ) £ E. Then 

(5.1) (e'(o),/i) =- 

( 2 -<z)||^IP 


for all {hi, ^, 2 ) £ E. 


Ta + 72 — 7i 

-2{2l-q) [ \w 1 nw 2 fdx' 
JQ 


Proof. For z = (Lyi,W 2 ) £ define a function : K x i? —^ R by 

J^z{C,p) ■■= - p)),£.{z - p)) 

= ^'^ks [ (|V(wi-pi)p + |V(w2-?32)P) 

Jca 

{\\wi-pi\'^ + p\w2-p2\'^)dx-2i^‘ [ \WI-Pi\°'\w2-P2fdx. 

Jn Jq 

Then J^(l, 0) = ^( 2 ), 2 :) = 0 and, by Lemma 4.3, we have 

d^(l, (0, 0)) ^ 2||^||2 _ ^ (a|wi| 9 + ^|ui2|'’) dx - 22* /" |uii|“|w2|^da; 

Jn Jn 




(2 - q)\\z\\^ - 2(2* -q) [ |u;i|“|w 2 |'^da; ^ 0. 
Jn 


In turn, by virtue of the Implicit Function Theorem, there exists r > 0 and a function f : B{0;r) G E 
of class such that ^(0) = 1 and formula (5.1) holds, via direct computation. Moreover, Jifz{^{h), h) = 0, 
for all h £ B{0;r), is equivalent to 

{JJ'\A^(.h){z - h)),^{h){z - h)) = 0 , for all h £ . 8 ( 0 ; r), 
namely ^(h)(z — h) £ A/a,□ 


Lemma 5.2. Let {X, p) £ “^Ai- Then, for each z £ A/)^ there exists r > 0 and a differentiable function 
: B{0;r) C E ^ 1R+ such that ■C~(0) = Ij ^~{k)iz — h) € A/(("^ for every h £ B{0;r) and formula 
(5.1) holds. 


Proof. Arguing as for the proof of Lemma 5.1, there exists r > 0 and a differentiable function : 
B{0;r) C E ^ R+ such that ^“(0) = 1, ^~{h){z — h) £ A/a.^i for all h £ B{0;r) and formula (5.1) holds. 
Since 

{J^'\.i^{z),z) = {2-q)\\zf-2{2*-q) ( \wi\°‘\w 2 \^dx < 0, 

Jn 

by the continuity of the functions TV), ^ and up to reducing the size of r > 0, we get 

{n'x.,,{S,-{h){z - h)),C{h){z - h)) = (2 - g)||r(/^)(^ - h)f 

-2{2t-q) [ \{r{h){z-h))in{r{h){z-h)hfdx<o, 

Jn 

where (^“(/i)(z — h))i £ XQ(Cn) denote the components of ^~(h)(z — h). This implies that the functions 
^~{h){z — h) belong to J^ff □ 

Proposition 5.3. The following facts hold. 

(i) Let (A,/i) £ taAi. Then there is a {PS)ax,,j,-sequence {zn} C Jffx,n forIx,fi. 

{ii) Let (A,/i) £ ‘^(„/2)2/(2“?)Ai ■ Then there is a {PS) - -sequence {zn} C A/T forXx,^. 

V'l/ / 


Proof, (i) By Lemma 4.1 and Ekeland Variational Principle [17], there exists a minimizing sequence 
{zn} C Mx.ii such that 

Tx,fi{Zn) ^ T ; 

n 

Tx,fi{zn) < Tx,it{w) + -\\w - Zn\\, for each w £ A/'a,^. 


(5.2) 




14 


X. HE, M. SQUASSINA, AND W. ZOU 


Taking n large and using a\^fi < 0, we have 

(5-3) = (^ - ^) ^i\w2,n\‘^)d3L 


< ClA,a H-< 


1 , «A„ 


This yields that 
(5.4) 


<72* 


2(2*-9) 


Jn 


Consequently, z„ 7 ^ 0 and combining with (5.3) and (5.4) and using Holder inequality 


IknII > 

and 

(5.5) IknII < 

Now we prove that 


< 72 : 


q 2 2 q-2 


2(2* - <7) 




2(2*-9),^,^Izi 


q 2 2 2-g 


< 7 ( 2 * - 2 ) 


|r 2 | {ksS{s,N)) ^ {X^-1 + 2 


l| 2 ^A,a( 2 n)||E-i -t 0, asn^oo. 


Fix n € N. By applying Lemma 5.1 to z„, we obtain the function ■ .6(0; r„) —R+ for some rn > 0, 
such that ^ra(7i)(z„ — h) G A/a,^. Take 0 < p < r^. Let w G E with w ^ 0 and put h* = .j^. We set 
hp = ^n{h*){zn — h*), then hp G ■Afx,^, and we have from (5.2) that 


^A,/i(hp) l-X^pi^Zu) ^ ll^p ^ri\\- 


By the Mean Value Theorem, we get 

(d^xA^n), hp - Zn) + o{\\hp - Zn\\) > --\\hp - 2„||. 

Thus, we have 

{d^'xA^n),-h*) + i^nih*) - l){Ix^p{Zn),Zn - h*) > -^\\hp - Zn\\ + o{\\hp - Z„ 
Whence, from ^n{h*){zn — h*) G M\^p, it follows that 

-p(i;^^(z„), + [Uh*) - m'xA^u) - T'x^pihp), Zr. - h*) 


>- \\hp - Zn\\ + o{\\hp - Zn 


So, we get 
(5.6) 


{l'xAZn),^)<-\\hp-Zn\\ + 
\ WwW / no 


0{\\hp - Zn\\) 


|u;||/ np 


{^n{h ) 1) ^ T-/ Ij, \ _ u*\ 


Since ||7ip - Zn\\ < p\^n{h*)\ + \^n{h*) - l|||^n|| and 

1 ™ ^(^*)-i| <ll^;(o)l|. 

p^O p 

For fixed n G N, if we let p —>■ 0 in (5.6), then by virtue of (5.5) we can choose a constant C > 0 
independent of p such that 

(^C(-'..).=)s2(i + |l&(o)ll). 
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Thus, we are done once we prove that ||C^i(0)|| remains uniformly bounded. By (5.1), (5.5) and Holder 
inequality, we have 

Ci\\h\\ 


|(c;(o),/i)| < 


(2 - g)||z„||2 - 2(2* - g) [ \wi^n\°‘\w2,nfdx\ 

Jq 

for some Ci > 0. We only need to prove that 


{2 - q)\\z.n\\'^ - 2{2* - q) f \wi^n\°‘\w 2 ,nfda 

Jn 


>C2, 


for some C2 > 0 and n large enough. We argue by contradiction. Suppose that there exists a subsequence 
{zn} such that 

(5.7) {2 - q)\\Zn\\^ - 2{2l - q) [ \wi,nnW 2 ,nfdx = On{l). 

Jo, 

By virtue of (5.7) and the fact that we have 

„ ■ [ \'Wl^n\°‘\w 2 ,nfdx + On{l), lkn|P = ^|- X,p,{^n) + On{l) ■ 

^ — <1 Jn — z 

Taking into account that Ix^^{zn) —>■ < 0 as n —?► 00, we have ||z„|| 74 0 as n —?► 00. Then, arguing 

as in the proof of Lemma 4.3 yields (A,/r) ^ “^Ai, a contradiction. Then, 

. w . C 




ll^ll n 

This proves (i). By Lemma 5.2, one can prove (ii), but we shall omit the details here. 


□ 


6. Local minimization problems 

Now, we establish the existence of a local minimizer for Ta,/x in 

Proposition 6.1. Let {X, iJ,) £ ‘^Ai- Thenlx,fj, has a local minimizer z~^ inJ\f^^ satisfying the following 
conditions: 

(i) Xa,^(z+) = ax,f, = < 0; 

(ii) z'^ is a positive solution of (2.1). 

Proof. By (i) of Proposition 5 . 3 , there exists a minimizing sequence {zn} = {iwi^n,W2,n)} for Ix,^i in 
A/a, 71 such that, as n —>■ 00, 

(6.1) Ix.fiizn) = ax,^^ + On{l) and l)^_^(2„) = o„(l) in E~^ . 

By Lemma 4 . 1 , we see that Ia./^ is coercive on Aa,^, and {Zn} is bounded in E. Then there exists a 
subsequence, still denoted by {Zn} and z~^ = {wf,W2) £ E such that, as n —>■ 00, 

{ wi,„ ^ W2,n^w:^, weakly in Ap(n), 

iPi,n —>• Wi", W2,n —>■ W2, strongy in L^{Ll) for all 1 < r < 2*, 
iPi,n —>• W2,n^W2, a.e. in fl, 

up to subsequences. This implies that, as n ^ 00, 

(6.2) Qx,tJ.{Zn) = Qx,^iiz~'') + On{l). 

We claim that z"*" is a nontrivial solution of ( 2 . 1 ). It is easy to verify that z'^ is a weak solution of ( 2 . 1 ). 
From Zn £ ffx,fi and ( 4 . 2 ) we deduce that 

( 6 . 3 ) Qx,M = 

Let n ^ 00 in ( 6 . 3 ), by ( 6 . 1 ), ( 6 . 2 ) and oa./x < 0 , we have 

Qa.m(^+) > > 0. 
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Therefore, G Afx.fj. is a nontrivial solution of (2.1). Now we show that —>• strongly in E and 

= aA./j- Since G A/a,^, then by (6.3), we obtain 






+ l|2 , 


<72* 


QxA^^) 


= lim Ix.fj,{Zn) = Oa./x- 

n—^oo 


This implies that = ax,^i and lim„_).oo W^nW^ = Since 

\\Zn-Z+r = \\Zr.r-\\z+r+oM), 

we conclude that z„ —>■ z+ in E. We claim that z+ G Assume by contradiction that 2+ G 

Then, by Lemma 4.5, there exist (unique) A with z~^ G -^Xfj, tiZ~^ G J^x^^■ particular, 

we have tf < = 1. Since 

^Ix,A^A\t=t+ = 0> and ^IxAt^A\t=t+ > 
there exists A < < ti such that Ix,tJ.{ti < ^x,tj.{t* z~^). By Lemma 4.5, we have 

ax,p. <Ix,Ai^^) <AA*^A <'^x,Ai ='^x,A^A =aA,/i, 

a contradiction. Since Ix^A^A = |wj|) and (|ii’i’|, Iw^l) G Mx,tJ,, by Lemma 4.2 we may 

assume that is a nontrivial nonnegative solution of (2.1). In particular wf ^ 0,r(;^ ^ 0. In fact, 
without loss of generality, we assume by contradiction that wj = 0. Then wf is a nontrivial nonnegative 
solution of 

—div(2/^“^®Vw) = 0 in Cn 
w = 0 on OlCq 

< 

w = u on n X {0} 

J^ = X\w\‘>-^w onnx{ 0}. 

By the maximum principle [27] we get wf > 0 in Xg(Cn) and 

||«,0)f = Q;,,^«,0) >0. 

Moreover, we can choose £ ArQ(^n)\{0} such that 

||(0,u;2*)f =Qa.m( 0 ,i« 2*)>0. 


Note that 

Qx,tJ.At^^2) = Qx,tJ.At:A + <3A.,i(0,W2) > 

and so by Lemma 4.5 there is unique 0 < <t < t~ such that (t+w)*', t+inj) G ■Afxfj,- Moreover, 


f At - AQ^A^i ,wi) 

\ (2*-2 )||«,u; 2*)P 


1 

2-q 



> 1 


and 

Ix,A~'''>^i^Aw 2 )= inf Ix,/j.itwAtw 2 ). 

0<t<t- 

This implies that 


<IxAwt,w;) <AA'^iA) =at,t, 

which is a contradiction. Then by the Strong Maximum Principle [10, Lemma 2.4], we have 
in Cn, hence, 2+ is a positive solution for (2.1). 


> 0 
□ 
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Next we will use We = Es{ue), the family of minimizers for the trace inequality (2.4), where Ue is given 
in (2.5). Without loss of generality, we may assume that 0 £ fl. We then define the cut-off function 
(j) G C“(Co), 0 < ^ < 1 and for small fixed p > 0 , 

1 , {x,y) € Bp, 

0 , {x,y)^B2p, 

where Bp = {(x, y) : |xp +y'^ < p^,y > 0}. We take p so small that B 2 p C Cq. Recall W is the extension 
of U introduced in Section 2, we have (cf. [3]) \VW{x,y)\ < Cy~^yV{x,y). Let 

1 


(/)(x, y) = 


Ue{x) = 

Then the extension of Ue{x) has the form 


(e 2 + |a;| 2 )^’ 


e > 0. 


>Ve(x,y) = Civ.s?/^® [ 

Jr 


Ue{z)dz 

R'^ (|a; — 2;|2 -|- ?/2 )-t^ 


= e 


2s-N 


W 




Notice that (()We G Xq{Cq), for e > 0 small enough. 

Lemma 6 . 2 . There is z € i?\{0} nonnegative and A* > 0 such that for (A,/r) £ “^a* 

svcpl\^p{tz) < 

Coo ; 


t>0 


where Coo is given in Lemma 3.3. In particular, ^ < Coo for all (A, p) £ “^a* • 
Proof. By an argument similar to that of the proof of [3, formula (3.26)], we get 

(6.4) umwh =ks [ y^-^^\VWe\^dxdy + 0(1) 


= e' 


+ 

2s-N 


We notice that 


ks [ ^®|V>V(x,y)pda;dy-hC>(l). 

r (j){xf* 

Jq 


UUeWti = / \m^^dx = 


\\Ue\\;i = 


IQ 


/ 

Jr^ 


1 


Then, one has that 




-N 


25 V (e^ + 

{x) — 1 


dx = e"^||17||2;. 


\mll = f 

Jn 


■dx — 


f 

Jw 


dx 


which yields 




L 

L 


a{e^ + \x?r JR^\n{e^ + \x?r^ 

dx 


1 

a\B(0;p) (^^ + kP)^ 
dx 


dx ■ 


K'^\B(0;p) + I^^P)^ Jr^\B{0;p) \x 


I 

Jr 

I 


R-\n + \x\^r 

dx 


\2N 


= c.. 


This implies that 


l-C:ie^\\U\\^:‘ <e^UU,r2l\\U\\^:‘ < 1 + ||C/||2p' 


Taking e so small that C 3 e^\\U\\ 2 * ^ < 1, since 2/2* = (A^ — 2s)/N < 1, we obtain 

l-e^C3||C/||2T < (1-£'^C3||C/||2TA"* 

< (l + £^C3||C/||2“5)"/"“ < 1 + £^C3||C/||2“5. 
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Hence H'/'HeHI. = Since W = Es{U) optimizes (2.4), by (6.4) we have 

^2s-Nks f y^~'^^\yyv{x,y)\^dxdy + 0(1) 


(6.5) 


UWefxS 

UUeWl e2.-A||f/||2^+0(e2«) 


ks [ ^®|VW(a:, y)pda:dy 


\mi, 

= ksS{s,N)+0{e^-^^). 


(l + 0 (e^- 2 ®)) 


Now we consider the function J : i? —>• R defined by J(z) := l/ 2 || 2 ;|p — 2/2*|ryi|“|it; 2 |^da:. Set 
i«o,i := ^/a(j)'We, Wo ,2 := '/P(j)'We and zq := (wo 4 ,wo, 2 ) & E. Notice that J(0) = 0, J(tzo) > 0 for t > 0 
small and J(tzo) < 0 for t > 0 large. The map 1 1 —>■ J(tzo) maximizes at 


/ 


( 6 . 6 ) 


to ■— 


INoll^ 




, 2 [ |wo,ir|wo,2|^da: 

\ Jn 


Then from (2.7), (6.5) and ( 6 . 6 ), we conclude that 


sup J(teo) = J{toZo) = ( X - 
t>o V ^ 


INolP 


s 

N 


{a + /3)ks [ ; 

JCa 


(2 j |wo,ir|wo,2|^) 


y ®|V((/>Ve)| dxdy 


^0 2/32 f \(f)Ue\'^‘^ dx'^ 

J O 


^ N-2s 

2^2r- 


(6.7) 


iV-2s 

N2 2 « 


N2- 


/3 




» + /3 


“ + '3 

a 


/3 


» + /3 


ks [ 2/1-2 ®|V(/>We)| 
JCn _ 

(^l^\^u,f^dxy 


hS{s,N)+Oie‘^-^^) 


2, N 


^ -rr^ N-2S 

N2^^ 


k,Ss,a,i3 + Oie^-^^) 


+ 0 (e"-“) = I ) + 0 (£''->-). 


We now choose > 0 so small that, for all (A, y) G ‘^ 5 ^, we get 

N 


Cqq — 


2s ( kcSc 


N 


- Ko{X^-i + ) > 0 . 


By the definition of TA,/i and zq, we have 

e 


'd^\,^l{tzo) < yll-Zor, for alH > 0 and A,/i > 0, 


which implies that there exists to G ( 0 , 1 ) satisfying 

sup Ix.f,{tzo) < 

^OO 7 

te[o,to] 


for all (A, y,) G 'i^Si ■ 
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Hence, from (6.7) and a,/? > 1 we see that 


( 6 . 8 ) 


suplA.M(teo) = sup (j{tzo) - —Q\,tj.{zo)) 


t>to 


t>to 


< 


N\ 2 J q \ ) Jb(o-,p) 


- N\ 

Letting 0 < e < p, we have 

[ lU.l’^dx = [ 

Jb(0:o) Je 


2 s, ^s^s,a,p\ 2s / /v_9c\ t| 


+ {X + p) 


1 


^(N-2s) dx > 


BiO-p) 

1 


(0;p) 

\Ue\‘‘dx. 


q{N-2s) 


■dx = C 4 , 


/s(0;p) (e2 + |a;p) 2 - 

for some C 4 = C' 4 (iV, s,p). Combining this with ( 6 . 8 ), for e = < p, 

9 


B(0;p) (2p2)- 2 " 

+ p^)'^- 

SUplA.p(teo) < +0 +M^) - —(A + ai)C'4. 

t>to Iy\Z/ \ / Q 


Choosing i 52 > 0 small enough, for all {X^p) G we have 

O (^X^ + + P-)Ci < -Ko ( a ^ + p^^ ■ 

If we set A* = minjdi, ^ 2 } > 0, then for (A, p) G , 

(6.9) SUplA,At(t^o) < Coo. 

t>o 

Finally, we prove that a'^ ^ < Coo for all (A, p) G . Recall that 

Zo = (wo,l,Wo, 2 ) = (\/a^We, \/8f/>We). 


By Lemma 4.5 there is to > 0 such that to^^o G By the definition of ^ and (6.9), we conclude 

Ta „ < 1 x,p{toZo) < S\XY>Tx,p{tZo) < Coo, 
t>o 

for all (A, p) G “^A* ■ n 

Let A* be as in Lemma 6.2. We prove the existence a local minimizer for Xx.p on Af^^. 

Proposition 6.3. Let A* > 0 be as in Lemma 6.2 and set 

A 2 := min{A*, {q/2)^Ai}. 

For (A,/i) G ‘^As; 21 a,^ has a minimizer z~ in with Ix,p{z~) = cxx p- Furthermore, z~ is a positive 
solution of (2.1). 

Proof. By (ii) of Proposition 5.3, there is a {PS) - sequence {z„} C AfS for Ix,p for all 

(A,/i) G ^(q/2)2/(2-<i)Ai ■ 

By Lemmas 3.3 and 6.2 and (ii) of Theorem 4.4, for (A, p) G I\,p satisfies the PS condition at the 
energy level af) ^ > 0. Therefore, there exist a subsequence still denoted by {zn} = {{wi^n,W 2 ,n)} G1 Afff^ 
and z“ := (ic)",w^) G E such that Zn —>■ 2 :“ strongly in E and 

(6.10) Xx,p{z~) = > 0 for all (A,/i) G ^a-^- 

Now we prove that G Afff By virtue of Zn G Afff^ we see that 

(7^'A^(z„),z„) = (2-g)||z„f- 2 ( 2 :-( 7 ) / |rci,„r|R; 2 .„|^ds < 0, Vn G N. 

Ja 
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Taking the limit in the last inequality and using > z in we get 

(6.11) {TZ'^^^{z-),z-) = {2-q)\\z-\\'^-2{2;-q) f < 0. 

in 

Observe that we must have the strict inequality in (6.11). Otherwise, by (6.10), we have z“ ^ (0,0), and 
so z“ G but this contradicts to Lemma 4.3. Thus, z“ G Since X\^^{z~) = X\^^{\z~\) with 

\z~\ = (|w(~|, Iw^l), and \z~\ G by Lemma 4.2 we may assume that z“ is a nontrivial nonnegative 

solution of (2.1). Moreover, by z“ G we get from (6.11) that 

^ Iw^nw^fdx > > 0 . 

This implies that ^ 0 ,W2 ^ 0. Using the maximum principle as in the end of the proof of Proposi¬ 
tion 6.1, we have > 0 in Cn. Hence, z“ G -Mx/j, ^ positive solution for (2.1). □ 

7. Proof of Theorem 1.1 concluded 

By Proposition 6.1, for (A, /r) G , system (2.1) has a positive solution z+ G J^Xfi- Proposition 6.3, 
a positive solution z“ G exists for (A, /i) G ^Ai. - Since = 0, then z=*= are distinct solutions 

of (2.1), so that (m^(x), u^(a;)) = (inj*^(a:, 0), w^(a;, 0)) are distinct positive solutions of (1.1). □ 

Acknowledgments. The authors would like to express their sincere gratitude to the anonymous referee 
for her/his careful reading the manuscript and valuable comments and suggestions. 
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